Skip to content

use half of the available cores for hipBLASLt - #281

Open
bedroge wants to merge 7 commits into
EESSI:mainfrom
bedroge:rocblas_parallel
Open

use half of the available cores for hipBLASLt#281
bedroge wants to merge 7 commits into
EESSI:mainfrom
bedroge:rocblas_parallel

Conversation

@bedroge

@bedroge bedroge commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

This should hopefully prevent it from running out of memory.

@bedroge

bedroge commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Let's give it a try

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-aws-eu-south on:arch=x86_64/amd/zen5 for:arch=x86_64/amd/zen5,accel=amd/gfx908+amd/gfx90a+amd/gfx942+amd/gfx1030+amd/gfx1100+amd/gfx1101+amd/gfx1200+amd/gfx1201

@eessi-bot-aws-eu-south

eessi-bot-aws-eu-south Bot commented Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown

New job on instance eessi-bot-aws-eu-south for repository eessi.io-2025.06-software
Building on: amd-zen5
Building for: x86_64/amd/zen5 and accelerator amd/gfx908+amd/gfx90a+amd/gfx942+amd/gfx1030+amd/gfx1100+amd/gfx1101+amd/gfx1200+amd/gfx1201
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_281/241

date job status comment
Aug 06 09:56:20 UTC 2026 submitted job id 241 awaits release by job manager
Aug 06 09:56:52 UTC 2026 released job awaits launch by Slurm scheduler
Aug 06 10:01:56 UTC 2026 running job 241 is running
Aug 07 10:03:23 UTC 2026 finished
🤷 UNKNOWN (click triangle for detailed information)
  • Job results file _bot_job241.result does not exist in job directory or reading it failed.
  • No artefacts were found/reported.
Aug 07 10:03:23 UTC 2026 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job241.test does not exist in job directory or reading it failed.

@bedroge

bedroge commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Not sure what's going on with these jobs. It doesn't look like they run out of memory, but it's getting stuck at some point, and ultimately Slurm aborts and requeues the job on the same node? On the node itself I only see these messages in the syslog:

[Thu Aug  6 11:11:51 2026] INFO: task bash:1680114 blocked for more than 122 seconds.
[Thu Aug  6 11:11:51 2026]       Not tainted 5.14.0-611.49.1.el9_7.x86_64 #1
[Thu Aug  6 11:11:51 2026] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Aug  6 11:11:51 2026] task:bash            state:D stack:0     pid:1680114 tgid:1680114 ppid:1680110 flags:0x00004002
[Thu Aug  6 11:11:51 2026] Call Trace:
[Thu Aug  6 11:11:51 2026]  <TASK>
[Thu Aug  6 11:11:51 2026]  __schedule+0x231/0x540
[Thu Aug  6 11:11:51 2026]  schedule+0x2c/0xb0
[Thu Aug  6 11:11:51 2026]  io_schedule+0x41/0x70
[Thu Aug  6 11:11:51 2026]  folio_wait_bit_common+0x11d/0x330
[Thu Aug  6 11:11:51 2026]  ? xas_init_marks+0x1f/0x50
[Thu Aug  6 11:11:51 2026]  ? __pfx_wake_page_function+0x10/0x10
[Thu Aug  6 11:11:51 2026]  invalidate_inode_pages2_range+0x179/0x3f0
[Thu Aug  6 11:11:51 2026]  fuse_open_common+0x1bf/0x220 [fuse]
[Thu Aug  6 11:11:51 2026]  ? __pfx_fuse_open+0x10/0x10 [fuse]
[Thu Aug  6 11:11:51 2026]  do_dentry_open+0x14c/0x440
[Thu Aug  6 11:11:51 2026]  do_open+0x219/0x400
[Thu Aug  6 11:11:51 2026]  path_openat+0x110/0x280
[Thu Aug  6 11:11:51 2026]  do_filp_open+0xb0/0x160
[Thu Aug  6 11:11:51 2026]  do_open_execat+0x5c/0xf0
[Thu Aug  6 11:11:51 2026]  open_exec+0x27/0x50
[Thu Aug  6 11:11:51 2026]  load_elf_binary+0x20c/0xe30
[Thu Aug  6 11:11:51 2026]  ? __kernel_read+0x18a/0x320
[Thu Aug  6 11:11:51 2026]  search_binary_handler+0xd2/0x250
[Thu Aug  6 11:11:51 2026]  exec_binprm+0x4f/0x1a0
[Thu Aug  6 11:11:51 2026]  bprm_execve.part.0+0x179/0x210
[Thu Aug  6 11:11:51 2026]  do_execveat_common.isra.0+0x1a0/0x220
[Thu Aug  6 11:11:51 2026]  __x64_sys_execve+0x33/0x40
[Thu Aug  6 11:11:51 2026]  do_syscall_64+0x5c/0xe0
[Thu Aug  6 11:11:51 2026]  ? xas_start+0x5f/0xd0
[Thu Aug  6 11:11:51 2026]  ? _raw_spin_unlock_irqrestore+0xa/0x30
[Thu Aug  6 11:11:51 2026]  ? next_uptodate_folio+0x85/0x260
[Thu Aug  6 11:11:51 2026]  ? _raw_spin_unlock+0xa/0x30
[Thu Aug  6 11:11:51 2026]  ? filemap_map_pages+0x45e/0x540
[Thu Aug  6 11:11:51 2026]  ? do_read_fault+0x10b/0x1d0
[Thu Aug  6 11:11:51 2026]  ? do_pte_missing+0x157/0x200
[Thu Aug  6 11:11:51 2026]  ? __handle_mm_fault+0x2fe/0x650
[Thu Aug  6 11:11:51 2026]  ? set_next_entity+0xda/0x150
[Thu Aug  6 11:11:51 2026]  ? __count_memcg_events+0x4f/0xb0
[Thu Aug  6 11:11:51 2026]  ? mm_account_fault+0x6c/0x100
[Thu Aug  6 11:11:51 2026]  ? handle_mm_fault+0x138/0x270
[Thu Aug  6 11:11:51 2026]  ? do_user_addr_fault+0x35d/0x620
[Thu Aug  6 11:11:51 2026]  ? exc_page_fault+0x61/0x150
[Thu Aug  6 11:11:51 2026]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[Thu Aug  6 11:11:51 2026] RIP: 0033:0x14f133880ce7
[Thu Aug  6 11:11:51 2026] RSP: 002b:00007ffe20441e68 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[Thu Aug  6 11:11:51 2026] RAX: ffffffffffffffda RBX: 00005562b7abfdb0 RCX: 000014f133880ce7
[Thu Aug  6 11:11:51 2026] RDX: 00005562b7ac3ac0 RSI: 00005562b7b052f0 RDI: 00005562b7abfdb0
[Thu Aug  6 11:11:51 2026] RBP: 0000000000000000 R08: 00007ffe20441e40 R09: 0000000000000000
[Thu Aug  6 11:11:51 2026] R10: 000014f1337bd9f8 R11: 0000000000000202 R12: 00000000ffffffff
[Thu Aug  6 11:11:51 2026] R13: 00005562b7b052f0 R14: 00005562b7ac3ac0 R15: 00005562b7b025d0
[Thu Aug  6 11:11:51 2026]  </TASK>

@bedroge bedroge mentioned this pull request Aug 7, 2026
Open
@bedroge bedroge changed the title use half of the available cores for rocBLAS use half of the available cores for hipBLASLt Aug 12, 2026
@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws on:arch=zen2 for:arch=x86_64/amd/zen2,accel=amd/gfx942

@eessi-bot-aws

eessi-bot-aws Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2 and accelerator amd/gfx942
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_281/186051

date job status comment
Aug 12 08:10:19 UTC 2026 submitted job id 186051 awaits release by job manager
Aug 12 08:11:17 UTC 2026 released job awaits launch by Slurm scheduler
Aug 12 08:12:22 UTC 2026 running job 186051 is running
Aug 12 09:01:16 UTC 2026 finished job id 186051 was cancelled
Aug 12 09:01:41 UTC 2026 finished
🤷 UNKNOWN (click triangle for detailed information)
  • Job results file _bot_job186051.result does not exist in job directory or reading it failed.
  • No artefacts were found/reported.
Aug 12 09:01:41 UTC 2026 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job186051.test does not exist in job directory or reading it failed.

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

bot:cancel jobid:186051

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

It did use make -j 8, but it still ran out of memory. Let's try with 1/4th of the cores.

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws on:arch=zen2 for:arch=x86_64/amd/zen2,accel=amd/gfx942

@eessi-bot-aws

eessi-bot-aws Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2 and accelerator amd/gfx942
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_281/186052

date job status comment
Aug 12 09:02:23 UTC 2026 submitted job id 186052 awaits release by job manager
Aug 12 09:02:45 UTC 2026 released job awaits launch by Slurm scheduler
Aug 12 09:09:48 UTC 2026 running job 186052 is running
Aug 12 10:08:15 UTC 2026 finished job id 186052 was cancelled
Aug 12 10:09:27 UTC 2026 finished
🤷 UNKNOWN (click triangle for detailed information)
  • Job results file _bot_job186052.result does not exist in job directory or reading it failed.
  • No artefacts were found/reported.
Aug 12 10:09:27 UTC 2026 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job186052.test does not exist in job directory or reading it failed.

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

bot:cancel jobid:186052

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Still OOM killed, there's a main process taking up a lot of memory, and 4 joblib workers/threads. This was just before the main process got killed:

  99987 bot       20   0   18,4g  18,1g   3924 S  93,7  58,5   3:33.54 python                                                                                                           
 120517 bot       20   0 2672228   2,5g  12768 R  98,2   8,1   7:50.11 python                                                                                                           
 120516 bot       20   0 2670180   2,5g   9152 R  99,1   8,1   8:04.04 python                                                                                                           
 120514 bot       20   0 1443808   1,3g   9256 R  92,8   4,3   7:26.91 python                                                                                                           
 120515 bot       20   0  221248 207544   9288 R  99,1   0,6   8:58.08 python

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Related issue: ROCm/hipBLASLt#1614

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Not sure how to deal with this, let's give it one more try with only 2 cores.

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws on:arch=zen2 for:arch=x86_64/amd/zen2,accel=amd/gfx942

@eessi-bot-aws

eessi-bot-aws Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2 and accelerator amd/gfx942
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_281/186064

date job status comment
Aug 12 10:24:03 UTC 2026 submitted job id 186064 awaits release by job manager
Aug 12 10:25:14 UTC 2026 released job awaits launch by Slurm scheduler
Aug 12 10:26:26 UTC 2026 running job 186064 is running
Aug 12 11:21:17 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-186064.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen2-accel-amd-gfx942-17865334710.tar.zstsize: 33 MiB (35321962 bytes)
entries: 21160
modules under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942/modules/all
Boost/1.88.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
hipBLAS-common/1.2.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
msgpack-cxx/7.0.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
Tensile/4.43.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
software under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942/software
Boost/1.88.0-rocm-compilers-19.0.0-ROCm-6.4.1
hipBLAS-common/1.2.0-rocm-compilers-19.0.0-ROCm-6.4.1
msgpack-cxx/7.0.0-rocm-compilers-19.0.0-ROCm-6.4.1
Tensile/4.43.0-rocm-compilers-19.0.0-ROCm-6.4.1
reprod directories under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942/reprod
Boost/1.88.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_104546UTC
hipBLAS-common/1.2.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_102632UTC
msgpack-cxx/7.0.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_104607UTC
Tensile/4.43.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_104755UTC
other under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942
no other files in tarball
Aug 12 11:21:17 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ OK ] (1/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025-foss-2024a-kokkos %scale=1_node /ade8cad7 @BotBuildTests:x86-64-zen2+default
P: perf: 428.602 timesteps/s (r:0, l:None, u:None)
[ OK ] (2/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025_update4-foss-2025b-kokkos %scale=1_node /e121eb9c @BotBuildTests:x86-64-zen2+default
P: perf: 434.537 timesteps/s (r:0, l:None, u:None)
[ OK ] (3/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86-64-zen2+default
P: latency: 1.4 us (r:0, l:None, u:None)
[ OK ] (4/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86-64-zen2+default
P: latency: 2.09 us (r:0, l:None, u:None)
[ OK ] (5/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86-64-zen2+default
P: latency: 0.18 us (r:0, l:None, u:None)
[ OK ] (6/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86-64-zen2+default
P: bandwidth: 7876.95 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 6/6 test case(s) from 6 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-186064.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@bedroge

bedroge commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Now the job itself failed with a MemoryError. Just before it failed, I saw three Python processes taking up >90% of the memory. Final attempt with a single core...

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws on:arch=zen2 for:arch=x86_64/amd/zen2,accel=amd/gfx942

@eessi-bot-aws

eessi-bot-aws Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2 and accelerator amd/gfx942
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_281/186065

date job status comment
Aug 12 11:19:46 UTC 2026 submitted job id 186065 awaits release by job manager
Aug 12 11:20:11 UTC 2026 released job awaits launch by Slurm scheduler
Aug 12 11:21:15 UTC 2026 running job 186065 is running
Aug 12 15:06:40 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-186065.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen2-accel-amd-gfx942-17865468650.tar.zstsize: 33 MiB (35387896 bytes)
entries: 21160
modules under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942/modules/all
Boost/1.88.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
hipBLAS-common/1.2.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
msgpack-cxx/7.0.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
Tensile/4.43.0-rocm-compilers-19.0.0-ROCm-6.4.1.lua
software under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942/software
Boost/1.88.0-rocm-compilers-19.0.0-ROCm-6.4.1
hipBLAS-common/1.2.0-rocm-compilers-19.0.0-ROCm-6.4.1
msgpack-cxx/7.0.0-rocm-compilers-19.0.0-ROCm-6.4.1
Tensile/4.43.0-rocm-compilers-19.0.0-ROCm-6.4.1
reprod directories under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942/reprod
Boost/1.88.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_141700UTC
hipBLAS-common/1.2.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_135659UTC
msgpack-cxx/7.0.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_141721UTC
Tensile/4.43.0-rocm-compilers-19.0.0-ROCm-6.4.1/20260812_142214UTC
other under 2025.06/software/linux/x86_64/amd/zen2/accel/amd/gfx942
no other files in tarball
Aug 12 15:06:40 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ OK ] (1/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025-foss-2024a-kokkos %scale=1_node /ade8cad7 @BotBuildTests:x86-64-zen2+default
P: perf: 437.379 timesteps/s (r:0, l:None, u:None)
[ OK ] (2/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025_update4-foss-2025b-kokkos %scale=1_node /e121eb9c @BotBuildTests:x86-64-zen2+default
P: perf: 439.962 timesteps/s (r:0, l:None, u:None)
[ OK ] (3/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86-64-zen2+default
P: latency: 1.37 us (r:0, l:None, u:None)
[ OK ] (4/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86-64-zen2+default
P: latency: 2.09 us (r:0, l:None, u:None)
[ OK ] (5/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86-64-zen2+default
P: latency: 0.19 us (r:0, l:None, u:None)
[ OK ] (6/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86-64-zen2+default
P: bandwidth: 7734.86 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 6/6 test case(s) from 6 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-186065.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bedroge